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This  report  pertains  to  “Exploiting  Decision  and  Policy  Models  More  Effec¬ 
tively  through  Intelligent  Search,  Data  Mining,  and  Visualization,”  funding 
numbers  DAAH04-1-0391. 


1  List  of  Manuscripts 

•  “On  Heuristic  Mapping  of  Decision  Surfaces  for  Post- Evaluation  Anal¬ 
ysis.”  Completed  September  30,  1996;  appeared  in  Proceedings  Hawaii 
International  Conference  on  System  Sciences  (a  refereed  conference), 
IEEE  Press  (published  electronically  on  CDROM),  January  1997. 

•  “On  Relevance  and  Two  Aspects  of  the  Organizational  Memory  Prob¬ 
lem,”  Garett  Dworman,  Steven  0.  Kimbrough,  Stephen  Kirk,  and  Jim 

R.  Oliver.  Version:  orgmems5  (December  15,  1997).  technical  report. 

•  “MOTC:  An  Aid  to  Multidimensional  Hypothesis  Generation,”  by  K. 
Balachandran,  J.  Buzydlowski,  G.  Dworman,  S.O.  Kimbrough,  T.  Shafer, 
and  W.  Vachula,  in  Nunamaker  and  Sprague,  eds.,  Proc.  Thirty-First 
Hawai’i  Int’l.  Conf.  on  System  Sciences,  IEEE  Computer  Press,  1998. 

•  “MOTC:  An  Interactive  Aid  for  Multidimensional  Hypothesis  Genera¬ 
tion,”  Balachandran,  K.,  Buzydlowski,  J.,  Dworman,  G.,  Kimbrough, 

S. ,  Shafer,  T.  and  Vachula,  W.  Journal  of  Management  Information 
Systems,  16  no.  1  (Summer  1999),  pp.  17-36.  File:  motcjmis5.pdf. 

•  Pattern- Oriented  Access  to  document  Collections,  University  of  Penn¬ 
sylvania,  Ph.D.  thesis,  1999,  by  Garett  Dworman. 
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•  Tate  Shafer  (undergraduate  student) 

•  Garett  Dworman  (Ph.D.  student) 

•  Balachandran,  K.,  Buzydlowski,  J.,  and  Vachula,  W.  (Ph.D.  students) 


3  Inventions 

None  during  this  period.  Subsequently  (winter  1999-2000)  two  inventions, 
disclosed  to  the  University  of  Pennsylvania  Center  for  Technology  Transfer, 
pertaining  to  fast  creation  of  indexes  and  to  incremental  updating  of  index 
files. 

4  Scientific  Progress  and  Accomplishments 

4.1  Background 

Candle-Lighting  Analysis  (CLA)  is  the  name  for  a  series  of  techniques  for 
automated  support  for  intelligent  examination  of  formal  models,  particularly 
management  science  models.  Further,  these  same  techniques  are  valuable 
for  validation  of  such  models.  Such  examinations  are  well-suited  for  model 
validation  and  for  post-evaluation  analysis,  and  can  be  particularly  useful 
for  providing  decision  makers  with  insight  into  the  system  being  modeled. 
Candle-lighting  analysis  holds,  and  has  demonstrated,  much  promise  for  re¬ 
ducing  users’  time-to-insight  as  they  work  with  decision  and  policy  models. 

The  primary  alternative  to  candle-lighting  analysis  is  manually-directed 
what-if  questioning.  Such  what-if  analysis  is  a  cumbersome  process,  appro¬ 
priate  mainly  for  small-scale  models;  it  is  a  process  forever  at  risk  of  be¬ 
ing  undermined  by  human  biases  and  cognitive  limitations.  While  valuable, 
manually-directed  what-if  questioning  is  widely  recognized  to  have  serious 
limitations,  yet  until  the  development  of  candle-lighting  analysis  concepts 
and  the  CLAP-NT  software  system,  there  existed  no  general,  principled, 
analyst-friendly  way  to  systematically  and  thoroughly  perform  what-if  ques¬ 
tioning. 

Candle-lighting  analysis  concepts  hold  the  promise  of  greatly  enhancing 
the  value  and  effective  use  of  management  science  models  in  the  U.S.  Army, 
and  DoD  generally.  Candle-lighting  analysis  modules,  operating  in  a  real¬ 
time,  distributed  environment,  have  an  essential  role  to  play  in  supporting 
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decision  making  in  a  wide  variety  of  functional  areas — including  logistics, 
personnel,  and  operations. 

Essential  to  the  concept  of  candle-lighting  analysis  (CLA)  is  the  creation 
of  multiple  databases  containing  information  gleaned  from  particular  formal 
or  mathematical  models  (including,  but  not  limited  to,  management  science 
models)  using  intelligent  heuristics  (including,  but  not  limited  to,  genetic 
algorithms).  Very  roughly,  CLA  treats  models  as  (creating  or  entailing) 
complex,  hyperdimensional  structures  in  decision  space.  The  resulting  (hy¬ 
perdimensional)  surfaces  need  to  be  explored  intelligently  and  mapped  for 
human  decision  makers.  These  explorations  yield  large  data  sets,  which  are 
stored  in  a  database  environment.  The  databases  are  then  mined  in  order  to 
give  decision  makers  insights  in  the  problem  at  hand,  and  to  suggest  promis¬ 
ing  policy  options  to  them. 

In  short,  CLA  works  with  decision  models,  which  decision  makers  use 
to  estimate  the  consequences  of  making  decisions.  Typically,  these  models 
apply  for  problems  of  optimal  allocation  of  resources.  In  such  situations, 
decision  makers  need — and  require — more  than  a  simple  recommendation 
regarding  the  “best”  alternative  course  of  action.  Faced  with  a  series  of 
possible  courses  of  action,  we  may  characterize  the  face  of  decision  as  a 
multidimensional  hypersurface  representing  the  quality  of  the  outcomes  for 
the  various  possible  courses  of  action.  Ideally,  decision  makers  (or  the  staff 
supporting  them)  require: 

1.  A  comprehensive  map  of  the  (very  complex)  decision  surface  presented 
to  them  by  the  choices  at  hand. 

2.  Tools  and  concepts  for  interpreting  and  visualizing  this  decision  surface, 
in  a  way  that  is  conducive  to  effective  decision  making. 

Part  1  of  this  project  was  aimed  at  developing  concepts,  principles,  and  usable 
software  for  meeting  these  requirements.  Part  2  was  aimed  at  developing 
effective  information  retrieval  concepts,  especially  as  they  pertain  to  candle¬ 
lighting  data.  Here  the  project  took  a  surprising  and  very  successful  direction. 
Broadly  we  developed  the  mathematics  of  the  DCB  representation  and  we 
extended  it  to  use  in  pattern-oriented  retrieval.  These  accomplishments  are 
documented  in  the  aforementioned  papers  and  technical  reports. 
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4.2  Summary  of  Accomplishments 

1.  Redesign  and  implementation  of  the  CLAP-NT  (Candle-Lighting  Anal¬ 
ysis  Program  in  NT). 

The  work  under  this  contract  continues  work  previously  supported  by 
the  Army  AI  Center.  Our  previous  work  produced  various  key  concepts 
for  candle-lighting  analysis  and  a  demonstration  prototype.  That  pro¬ 
totype  yielded  important  proof-of-concept  demonstrations,  but  time 
and  resource  contraints  prevented  it  from  being  implemented  in  a  fully 
general  fashion.  (For  example,  it  was  limited  to  handling  only  about 
30  models.) 

During  this  phase  of  the  contract  we  performed  considerable  redesign 
and  implementation  of  the  CLAP-NT  system.  While  much  of  the  “look 
and  feel”  remained  constant,  the  internal  design  was  changed  radically. 
As  a  consequence  of  this,  for  example,  the  database  design  is  now  quite 
general  and  can  handle  an  indefinitely  large  number  of  models.  Also, 
models  are  now  implemented  as  objects.  This  considerably  simplifies 
integrating  new  models  into  the  system  and  allows  for  use  of  distributed 
models,  models  residing  on  other  computers  and  accessed  over  the  In¬ 
ternet,  for  example. 

2.  Location,  evaluation,  and  implementation  of  a  series  of  new  models  in 
the  CLAP-NT  system. 

The  most  important  of  these  models  are  described  in  our  paper  (see 
the  paper  cited  in  §1).  We  think  that  the  range  (e.g.,  including  integer 
programming  models,  knapsack  models,  goal  programming  models)  and 
scope  of  these  models  (e.g.,  applications  in  logistics,  federal  funding 
allocations,  and  construction  design)  amply  demonstrate  the  usefulness 
of  CLA  concepts  and  the  CLAP-NT  system. 

3.  Implementation  of  additional  heuristic  search  methods. 

At  the  start  of  this  contract  we  had  three  heuristic  search  methods 
for  generating  the  map  of  the  decision  surface  (see  requirement  (1)  in 
§4.1):  search  by  genetic  algorithm,  random  search  (for  benchmarking 
purposes),  and  greedy  hill-climbing.  During  this  period  we  reimple¬ 
mented  these  methods  as  objects  (see  item  (1),  above)  and  we  added 
new  methods  (as  objects):  simulated  annealing  and  branch-and-bound. 
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4.  Performance  of  computational  experiments. 

(These  are  partly  reported  in  our  paper,  see  §1.) 

5.  Perform  research  and  analysis  on  (a)  what  is  known  about  the  structure 
of  decision  surfaces,  and  (b)  general  statistical  methods  for  KDD  in  the 
context  of  CL  A. 

5  Technology  Transfer 

None  during  this  period,  but  Practical  Reasoning,  Inc.,  of  Bala  Cynwyd,  PA, 
a  recent  startup,  will  be  using  these  techniques  extensively. 
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